Field of the Invention
The present invention relates to an image processing apparatus that performs image restoration processing and an image processing method, and more particularly to processing for correcting a degraded image.
Description of the Related Art
In general, when obtaining an image by photographing an object using an image pickup apparatus, such as a digital camera, the image is significantly degraded e.g. due to aberration of an image pickup optical system (i.e. the image is blurred). A blur of an image is generally caused by spherical aberration, coma aberration, field curvature, astigmatic aberration, or the like, of the image pickup optical system. In an aplanatic state without any influence of diffraction, a light flux from one point of the object converges to one point again on an image pickup surface of an image pickup device. On the other hand, if any of the above-mentioned aberrations exists, light, which should converge to one point again on the image pickup surface, diverges to generate a blur component on an image thus formed.
The blur component generated on the image is optically defined by a point spread function (PSF). Although an image which is out of focus is also blurred, here, a blur of an image caused by aberration of the image pickup optical system even if the image is in focus is referred to as the “blur”. As for color bleeding on a color image, color bleeding caused by axial chromatic aberration of the image pickup optical system, spherical aberration of color, and comatic aberration of color can be referred to as different manners of blurring dependent on wavelengths of light. Further, as for color shift in a horizontal direction of an image, color shift caused by lateral chromatic aberration of the image pickup optical system can be referred to as positional shift or phase shift caused by different image pickup magnifications dependent on wavelengths of light.
An optical transfer function (OTF) obtained by Fourier transform of the above-mentioned PSF is a frequency component of aberration, and is represented by a complex number. An absolute value of the optical transfer function (OTF) (hereafter, the “optical transfer function” is simply referred to as the “OTF” as deemed appropriate), i.e. an amplitude component is referred to as the modulation transfer function (MTF), and a phase component is referred to as the phase transfer function (PTF).
These MTF and PTF are frequency characteristics of, respectively, the amplitude component and the phase component of degradation of an image caused by aberration. The phase component is expressed as a phase angle by the following equation (1). Note that Re(OTF) and Im(OTF) express the real part and the imaginary part of the OTF, respectively:PTF=tan−1{Im(OTF)/Re(OTF)}  (1)
The OTF in the image pickup optical system degrades the amplitude component and the phase component of an image, and hence in the degraded image, points of the object are asymmetrically blurred e.g. in a case where the degradation is caused by comatic aberration. Further, in a case where the degradation is caused by lateral chromatic aberration, the image formation position is shifted due to an image formation magnification different between optical wavelengths, and when the light is received as the RGB color components according to spectral characteristics of light reflected from the object, this causes different image magnifications between the color components.
This causes shifts in image formation position not only between the red, green and blue (RGB) components, but also between the wavelengths in each color component. That is, the image is diverged by the phase shift. To be exact, the lateral chromatic aberration does not generate simple parallel color shift. However, description below will be given assuming that the color shift has the same meaning as the lateral chromatic aberration, unless otherwise specified.
As a method of correcting degradation in amplitude (MTF) and degradation in phase (PTF), for example, a method of correcting degradation using the OTF of the image pickup optical system is known. This method is referred to as image restoration or image recovery. In the following description, processing for correcting degradation of an image using the OTF of the image pickup optical system is referred to as image restoration processing or simply restoration processing.
Now, the outline of image restoration processing will be described. Let it be assumed that a degraded image is represented by g(x, y), the original image is represented by f(x, y), and the PSF obtained by performing inverse Fourier transform on the OTF is represented by h(x, y). In this case, the following equation (2) holds. Note that * represents convolution, and (x, y) represent coordinates on the image.g(x,y)=h(x,y)*f(x,y)  (2)
When the equation (2) is converted to a frequency-based form by Fourier transform, this gives a form of the product, on a frequency-by-frequency basis, as represented by the following equation (3). Note that H represents a result of Fourier transform of the PSF, i.e. the OTF, and G and F represent results of Fourier transform of the degraded image g and the original image f, respectively. Values of (u, v) represent coordinates of a point on a two-dimensional frequency surface, i.e. a frequency.G(u,v)=H(u,v)·F(u,v)  (3)
To obtain the original image from the degraded image obtained through photographing, it is only required to divide both sides of the equation (3) by H, as represented by the following equation (4):G(u,v)/H(u,v)=F(u,v)  (4)
By returning F(u, v) in the equation (4) by inverse Fourier transform to a real surface, it is possible to obtain the original image f(x, y) as a restored image.
Here, assuming that a result of inverse Fourier transform of 1/H in the equation (4) is represented by R, by performing convolution processing on the image on the real surface, as represented by the following equation (5), it is possible to similarly obtain the original image.g(x,y)*R(x,y)=f(x,y)  (5)
R(x, y) in the equation (5) is referred to as an image restoration filter. The actual image has a noise component, and hence if the image restoration filter generated by the reciprocal of the OTF is used as mentioned above, the noise component is amplified together with the degraded image, and as a result, it is impossible to obtain a good image.
To prevent the noise component from being amplified, for example, there has been proposed a method of suppressing a restoration rate of high-frequency components of an image according to an intensity ratio between the image and noise, as in the Wiener filter. Further, as a method of correcting degradation of an image, caused by a color bleeding component, there has been proposed a method of correcting the color bleeding component by correcting the above-mentioned blur component such that the amount of blur is uniform for each of color components of the image.
The OTF changes according to the photographing state, such as a state of a zoom position, and a state of an aperture diameter. Therefore, the image restoration filter used in image restoration processing is also required to be changed according to the photographing state. For example, in an endoscope for observing an inside of a living body, there has been proposed a method of eliminating a blur of an image in a range outside an in-focus range of an image pickup section, using the PSF according to a fluorescent wavelength (see Japanese Patent Laid-Open Publication No. H10-165365). In this method, since the fluorescence is weak, an objective optical system having a small F-number is required. However, if the objective optical system having a small F-number is used, a depth of focus becomes shallow, and hence an in-focus image is obtained by performing image restoration processing for a range in which the object is out of focus.
As described above, image restoration processing is performed on an image obtained through photographing to thereby correct the above-mentioned various types of aberration, whereby it is possible to improve image quality. However, in performing photographing, the photographing state and the state of the image restoration filter do not always optimally match. For example, when photographing a three-dimensional object, such a problem occurs.
In the image pickup apparatus, photographing is performed by focusing on one surface of an object space using auto focus or manual focus. In doing this, in a case where the object is three-dimensional, the object distance is different depending on the angle of view. An object which is in focus is relatively sharply photographed, but an object which is out of focus is photographed with an amount of blur dependent on the distance. When information on the object distance is acquired only as to an in-focus point, an image restoration filter optimum for each angle of view in this object distance is selected or generated for use.
On an image after being subjected to image restoration processing, the image restoration filter is optimum for an object which is in focus, and hence it is possible to obtain desired sharpness. On the other hand, the image restoration filter is not optimum for an object which is out of focus, and hence although some effect of restoration is obtained, the image is still blurred.
On the other hand, it is conventionally known that a degree of blur dependent on the object distance produces excellent effects in expressing three-dimensionality of an object or expressing an object being watched in isolation from its background. For example, by using a telephoto lens with a shallow depth of field, an image is expressed such that a main object is in focus and the background is intentionally blurred. In this case, also on the image after being subjected to image restoration processing, it is desirable that the object which is in focus is made sharper, and the object which is out of focus remains still blurred, and blurring expression is performed by using the above-mentioned image restoration method.
However, if the object which is out of focus is subjected to image restoration processing using an image restoration filter which is not optimum for the distance of the out-of-focus object, coloring sometimes occurs on the image. Note that the term “coloring” refers to a defect that a color which is not included in the object is generated on the image after being subjected to image restoration processing because a relationship of blurring between the respective color components on edge portions of the out-of-focus object is different before and after execution of image restoration processing.
Further, such coloring sometimes occurs not only in photographing of a three-dimensional object. More specifically, coloring occurs irrespective of whether or not the object is in focus, if the aberration state in the actual photographing state and the aberration state targeted by the image restoration filter are different e.g. due to manufacturing variation of the image pickup optical system or variation of spectrum of a light source in photographing.
As a method of suppressing the coloring described above, for example, there has been proposed a method of correcting the color of an image after being subjected to image restoration processing based on color information on the image before being subjected to image restoration processing. In this method, a change in color, caused by image restoration processing, is determined for each pixel of the image to thereby suppress coloring caused by image restoration processing.
For example, there has been proposed a method of correcting a signal value so as to reduce an amount of color difference when the color difference in an image after being subjected to image restoration processing becomes larger than that before being subjected to image restoration processing (see e.g. Japanese Patent Laid-Open Publication No. 2010-86138).
As described above, by performing image restoration processing on an image obtained through photographing to reduce coloring which occurs e.g. on an image of an object which is out of focus, and correcting various types of aberration, it is possible to improve image quality.
However, in performing photographing, noise is generated during photoelectric conversion performed by the image pickup device, whereby a noise component is included in the image. In general, as the sensitivity of the image pickup device is set to be higher, this noise becomes larger. When coloring suppression processing is performed on the image including a lot of noise generated during photoelectric conversion, according to a color difference before and after being subjected to restoration processing, color tone of the object in the image is sometimes changed or inaccurate.
FIGS. 20A to 20E are diagrams useful in explaining pixel values of a G signal and an R signal and a color difference between the G and R signals along one line in an image including noise generated during photoelectric conversion performed by the image pickup device before and after being subjected to restoration processing. FIG. 20A shows changes in pixel values before restoration processing while FIG. 20B shows changes in pixel values after restoration processing. FIG. 20C shows changes in color difference before restoration processing while FIG. 20D shows changes in color difference after restoration processing. Further, FIG. 20E shows comparison of the color differences before and after restoration processing.
In this example, although changes in brightness and color of the object are small, as shown in FIG. 20A, the pixel value changes due to noise generated before image restoration processing. As mentioned hereinabove, it is desirable that the image restoration filter used in image restoration processing is formed by taking into account amplification of noise.
However, it is difficult to completely separate a degraded image degraded by aberration of the image pickup optical system and a noise component, and hence the noise is amplified by image restoration processing. That is, as shown in FIG. 20B, changes in pixel value are not reduced after image restoration processing. Note that in FIG. 20B, “R” represents an R signal value after applying the image restoration filter, and “R′” represents a signal value on which correction for suppressing coloring has been performed.
FIG. 20C shows the color difference before image restoration processing, and the color difference here refers to a difference (R−G) between the G signal and the R signal. FIG. 20D shows the color difference after image restoration processing, in which “Cr” represents the color difference after applying the image restoration filter, and “Cr′” represents the color difference on which correction for suppressing coloring has been performed. In this example, the R signal is corrected with respect to a pixel whose color difference is larger after being subjected to image restoration processing than before being subjected to image restoration processing, such that the amount of color difference is reduced.
FIG. 20E shows a moving average of the color difference for the purpose of comparison between the respective color differences before and after image restoration processing. Although in image processing performed by the image pickup apparatus, so-called color difference smoothing processing is performed, in this example, color tone of an image of an object in viewing the image is compared using moving averages as a simplified method.
As shown in FIG. 20E, when correction for suppressing coloring is performed, a change in the color tone occurs between before and after image restoration processing. Further, the change in the color tone shows a tendency made different by the amount of noise. That is, the amount of change in color tone is also changed depending on the sensitivity (ISO sensitivity) of the image pickup device.
However, the method described in Japanese Patent Laid-Open Publication No. 2010-86138 does not address a change in the color tone caused by the amount of noise, and hence it is difficult to properly suppress coloring caused by image restoration processing.